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Abstract 


ASSEMBLY  OnEBATIOiS 

This  paper  considers  some  of  the  qoeueing  problems  of  interest 
In  the  study  of  a  single  sewer  vho  assembles  two  items  to  produce  a 
single  unit  of  output.  This  systea  is  a  siapllfied  prototype  of  many 
industrial  operations.  Some  characteristics  of  optimal  assesbly 
operations  are  developed. 


The  system  to  be  considered  consists  of  an  operator  who  assembles 
two  parts  to  make  one  item  o£  finished  product.  The  traditional  single 
server  queue  can  provide  the  analysis  of  the  congestion  levels  of  one 
part  for  several  special  cases  of  this  system.   One  instance  of  this 
occurs  when  one  of  the  parts  is  always  available.   In  another  case,  one 
part  is  ordered  or  produced  after  each  assembly.   In  this  situation,  the 
time  to  produce  the  first  part  can  be  added  to  the  time  to  assemble  the 
item  to  give  an  occupation  time,  i.e.,  time  until  the  server  is  ready 
to  begin  the  next  assemt>ly.   In  another  special  situation,  one  of  the 
parts  for  the  next  assembly  is  ordered  or  produced  at  the  start  of 
each  assembly.   Using  an  occupation  time,  which  is  the  maximum  of  the 
assembly  operation  time  and  the  time  to  obtain  the  part  which  has  just 
been  ordered  for  the  next  assembly,  allows  the  standard  queueing  results 
to  be  used. 

There  are  many  questions  about  this  general  type  of  system  which 
can  be  raised,   Tltis  suggests  that  the  first  consideration  should  be 
to  determine  which  systems  are  in  some  sense  good.   From  a  management 
science  point  of  view,  the  study  of  optimal  designs  and  controls  should 
be  the  goal  of  the  study  of  congestion  systems.   For  the  system  considered 
here  it  is  almost  necer^'s.ry  t>  begin  with  thase  questions.   The  ifeason 
is  that  if  the  differences  between  the  expected  number  of  arrivals  of 
the  two  parts  becoaies  arbitrarily  large  as  time  goes  to  infinity,  the 
system  must  be  unstable.  Most  real  systems  of  this  type  do  not  exhibit 
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this  marked  Instability,   Often  such  systems  are  components  of  larger 
systems  in  which  it  is  possible  to  control  the  arrival  processes  at 
least  partially.  The  function  of  queues  in  any  system  is  to  decrease 
the  dependence  among  the  production  or  service  operations  performed 
in  the  system.  Although  the  limiting  conditions  of  unbounded  queues  In 
networks  lead  to  simple  results  [l])  this  extreme  is  not  likely  to  be 
desirable  when  there  are  costs  of  providing  and  maintaining  waiting 
lines.  Unfortunately,  most  of  the  analyses  of  queueing  theory  depend 
on  the  assumption  of  only  one  reflecting  barrier  at  the'  system  empty 
condition.  The  combination  of  the  need  to  work  in  two  or  more  dimensions 
and  a  bounded  state  space  require  major  developments  in  an« lysis. 

A  Model 

A  simple  model  can  be  developed  for  the  two  dimensional  process 
N(t)  »  [N^(t),  Ng(t)]  V7ith  N^^(t)  the  number  of  parts  of  type  A  in  the 
system  and  M,,(t)  t'\e  number  cf  type  B  parts.  Assume  that  the  process 
operates  in  discrete  time  with  the  tirne  interval  si^all.  Assume  that  if  • 
there  is  a  f^  rt  of  type  A  and  also  one  of  type  B  in  the  system  at  the 

start  of  the  period,  there  is  a  probability  of  [j,  of  completing  the 

A 
assembly  in  i-c  period,^  Assume  that  there  ts  a  probability  X.  .(t)  of 

an  arrival  of  a  pe,;.  t  cf  fype  A  in  the  period  (t,  t-J-At)  if  N(t)  «=  (i,j). 

A 
Furthermore,  assume  that  Xjj (t)  may  be  chosen  to  have  any  value  between 

A  -a 

0  and  X  for  any  state  and  any  time.  Similarly,  XJ^(t)  may  be  chosen 

B 
between  0  and  X  .  Moreover,  assume  that  at  most,  one  event  occurs 
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in  a  period.  The  transition  probabilities  for  the  process  are 


ij  rs 


hi 


ij 


1 

(r.8)   -   (i-j,   .1-1) 

(r,a)  »  (i,j) 

(r,s)  «   (i+l,j) 

(r,8)  -   (i,j+l) 

otherwise 

where  e(ij)  » 


i  or  j  *  0 
i  and  j  >  1 


The  analysis  of  this  process,  even  for  reasonable  assumptions  on 
X^.(t),  Xfj(t)  is  a  formidable  challenge. 

A  Profit  Structure 

In  order  to  focus  on  optimal  systems,  it  is  necessary  to  assume  some 

sort  of  a  profit  structure.  A  simple  but  interesting  structure  is 

provided  by  assuming  a  cost  for  holding  parts  and  a  revenue  for  each 

item  produced.   Specifically,  for  each  item  of  type  A  stored  in  the 

system  for  a  period,  assume  the  cost  is  hx „  For  each  part  of  type  B, 

charge  h  per  item  per  period.  For  each  item  completed,  assume  that 
B 

A     B 
there  ia  a  gain  of  g.  Let  V^-CX^i,   X^^O  be  the  expected  one-period  profit 

starting  form  state  ij.  The  assumptions  imply  that:        ' 


,B 


U 


j(^ij«  ^ij)=-  i^A  i  "  ^B  J  -  A.2   \   -    4i  ^B  -^  ^^<8  +  ^A  +  hg)  e(i,j) 
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DynatRJc  Program 

The  obvious  analysis  for  the  problem  of  selecting  values  for 

A         B 
Xij(t)  and  ^ijCt)  is  dynamic  programming.  Since  the  one  period 

A       B 
expected  profit  is  linear  in  X^.  and  \j^   with  negative  coefficients,  the 

optimal  policy  for  one  period  is  to  use  zero  arrival  probabilities  for 

all  states.   The  recursive  problem  has  the  form 


Wni.j  =  MAX  3   ^v(4.,  ^p3+  m[)^y    Xl^h   W^:,} 
■ij,  iji 
where  Wj^^  \    ^^   ^^^  maximum  discounted  expected  profit  from  n  periods  start- 
ing from  state  ij,  and  Wj^  is  the  matrix  of  these  quantities,  p  is  the 
discount  factor,  and  T  is  the  tensor  whose  elements  are  the  transition 
probabilities.  Clearly,  the  objective  function  is  always  linear  in  the 
decision  variables,  and  thus  they  must  be  either  0  or  their  maximum  values, 
The  same. is  true  for  the  infinite  horizon  problem  assuming  that  it  is 
meaningful.  Thus  the  decision  problem  really  is  equivalent  to  one  in 
which  fi'hara  are  four  choices  available  in  each  state. 

Properties  of  J).-,  J^1^''' ,^°^.  ,f^'9J:F^}:^^P\.:-.^y'^ 

The  obvicas  properrlas  co  try  to  establish  for  the  n  period  optimal 

A       A 
policy  are  that  the  optimal  Xij  is  }.     for  low  values  of  i  and  0  for  high 

values  of  i  for  each  j  artd  the  sjTnetric  result  for  Xj^a*     This  property 


-5- 


defines  two  single-valued  boundry  functions,  i^^-^^  ^^^   which  the  optimal 

X^i    is  0  for  i  >  i^Cj)  ^^^   Jn^^^  ^*^^  which  the  optimal  Xjj  is  0  for 

* 
j  >  Jn(i)«   It  should  not  be  surprising  that  these  functions  exhibit 

some  smoothness  properties.   First,  they  are  monotone  non-decreasing. 

The  somewhat  surprising  property  is  that  when  the  functions  increase, 

the  magnitude  of  the  increase  is  one.  Thus,  i^iO)  5  ^n  (J+1)  ^^^ 

i^(j)  +  1  >  in(j+l).   Similarly,  j*(i)  <  j*(i+l)  and  j*(i)  +  1  >  j*(i+l) 

This  last  characteristic  does  have  an  intuitive  explanation.   If  the 

optimal  policy  defines  a  transition  operate  which  makes  it  possible 

to  enter  state  i,j  from  one  state  k,m  "^  i  •»  j,  then  it  makes  it  possible 

to  enter  either  from  i-l,j  cr  l,j-l,  or  both,  which  is  clearly  the 

most  direct  possibility.  Thus,  the  optimal  n  period  policjr  has  the 

form  shown  in  Figure  I, 


*  y 


3  ifLy^iL,- jfL:^  '■»*■  ;«/ 
y    /    /^'  y«;»  ;™ 


0   I   2   3   4   5   i 

Figure  I  Optitr^al  20  period  policy  for 

A        B 
X  =  .1,  X  «  .2,  M,  =  .3,  h^  =  1,  hg  =  2,  pi  =  60,  3  =  .9 

states  i,J*Q(i)  are  in  O's,  states  i*(j),j  are  marked  by  x's.  The 

arrows  show  possible  transitions  of  the  optimal  operator.  Ergodic 

states  (CO),  (1,0),  (2,0),  (l,'!),  (2,1),  (2,2).  (3,2). 
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Properties  of  W 

To  prove  the  proper tiec  of  the  optimal  ti  period  policy  requires 

a  discussion  of  the   ^ropertie^a  of  the,   first  differences  of  the 

functions  W  .  The  optimal  policy  will  have  the  required  form  if 

a)  W  ^  ,  -  ^v^  ,  ,  .  is  non-increaaing  in  i  and  non-decreasing 
ni,j     ni-l,j 


in  J. 


b)  W  .  .   -  W  .  .  ,   is  non-increasing  in  j  and  non-decreasing 
ni^j.,;    ni^j-i 


in  i. 


""^      ^\i,j  "  ^ni-l,j^  1   ^^ni-l,j-l  "  ^ni.2,j-l) 
ni,j    ni,j-l'  —   ni-l,j-l  -  ni-l,j-2' 


If  these  properties  hold,  then  they  will  also  hold  for 
'h  ^  ^^ni,J  -  ^ni-lj>  ^"^  -  ^B  ^  ^<^ni,J  '  ^ni,M> 


.A 


which  are  the  two  test  criteria  which  determine  the  optimal  X. .  and 

p 
X..,  respectively.   Clearly,  property   a)  implies  that  i*,^(j)  will  be 

single  valued  and  non-decreasing  in  j.   Property  c)  quarantees 

A      A 
that  when  the  optimal  X. .  is  X  ,  then  . 

~    A      ni,j    ni-ljj'^  —    A  .      ni-l,j-l    ni,Z,j-l 

A         A 
and  the  optimal  X,  ,  ,  .  =  X  .   Property  d)  provides  this  same  property 

I  -  i  J  J  -  J. 

n 

for  the  optimal  X .  „. . 

I I 


' 


-7- 


Re  formulation 

To  prove  these  propartie??  of  the  functions  W  requires  examination 

of  the  TW  ,.   If  W  ,  v/ere  «  function  of  ona  variable,  it  might  be 
n- i       n-i 

possible  to  argue  that  T  ip  -4  toif..?lly  positive  operator  [3k] »  and  thus 

preserves  the  required  unirood^l  propertlea.  Unfortunately,  for  linear 

operators  applied  to  functions  of  two  veriables,  very  few  general 

results  are  available.  One  ciasb  of  operators  which  does  preserve 

the  montonicity  of  first  differences  is  the  class  of  positive  translation 

operators,   T  is  positive  translation  operator  on  functions  of  two 

integer  variables  if  the  coordinate  i,j  of  W  is  (TW).  .  «  E-  -2.   t    W^ 

i,j  reSj^sss^  r,a  i-r,j-! 

for  some  positive  numbers  t   ,  reS.  and  seS«.  For  such  an  operator,      ^ 

W,  ,  -  W,    ^   >  0  for  al  i  and  j,  then  TW  will  have  this  same 
i,j    i-n,j-m-- 

property.  Similarly,  if  W. .  -  W.    .   is  monotone  increasing  or  de- 
creasing  in  i  and  j,  TW  will  also  have  this  same  property. 

Unfortunately,  the  operators  in  this  problem,  even  with  the  regularity 
of  the  optimal  policy,  are  not  translation  operators.  The  fact  that  there 
are  four  regions  in  the  optiiiial  policy,  corresponding  to  all  four  choices 
of  the  two  arrival  probabilities  is  one  difficulty.  The  second  departure 
from  the  requiremctnts  of  a   translation  operator  occurs  along  the  boundary 
of  the  state  space.  When  i  or  j  is  zero,  then  there  is  no  assembly 
possible,  and  the  term  \.\,  "^ .    ,  .  .  is  not  present,  nor  is  -jji  W,  ,.  To 
permit  an   orderly  discussion  of  ail  the  special  cases  which  are  in- 
herently possible  in  TVJ,  a  reformulation  of  the  problem  can  be  developed 
which  avoids  making  the  boundary  state  special. 
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To  do  this  for  i,j  >  -1,-1,  define 


V 


i,j 


{ 


-(l.|i.)g  -  h^(j+l) 


i,J  >  0,0 


and 


W 


ni,j 


W 


W 


n  0,  j+1 
n  i-fi,0 


W 


n  i,j 


i=-l 
j=-l 
i,j>  0,0 


Further,   redefine  T  as 


/ 


i,j,k,tn  "    ^ 


X* 

x« 


x!j 


k,m  =  i-l.j'-r 
k,m  =  i;j 

k,m  =   i+l,j 

k,m  =   i,J+l 


Using  these  definitions, 
V  =  l^V* 

and  V  +.^TW  =  T  (V^  +  &W*  ) 

n  r. 

■k  "k 

The  function  V  +  3  W^  will  have  the  required  first  difference  properties 

if  W  has  them  arid   in  addition 
n 


*        * 

(V.  .  +  is  w 


lO      n  i,J      C,j      n  0,j^  ~  ^  G,j      n  Qj^     -i,j    "^  n  -l,j' 


and  the  corres ponding  inequality 


3  Cw  -  ,  -  2W    .  +  W    ..  J  <  S  +  h^  ^  h„ 
n  1,J     n  o,j    n  o,j+l  —  ^    A  +  B, 
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This  inequality  will  certainly  hold  if  it  holds  for  P*=l.  An  alternative 
form  of  this  inequality  can  be  found,  assuming  the  properties  of  W_ 
previously  mentioned  are  true.   In  particu,lar,  since 


the  previous  inequality  will  hold  if 


n  l,j    n  0,j-l  —  ^         A 


A  more  general  result  is 

e)  W   .  ,  -  W   .  .  .  .  <  g  +  h.  -I-  h_, 
'    n  1,  j    n  1-1,  j-1  —  =*    A    B' 

and  it  is  this  one  which  will  be  established  inductively.  This  in- 
equality will  also  supply  the  syraetic  condition  required  for  the 
boundary  on  which  j-O. 

Induction 

The  function  W  ..  =  0  clearly  satisfies  all  of  the  properties, 

oij 

a  to  e.  Within  each  of  the  four  rej^ions  of  the  optimal  policy,  the 
optimal  T,  is  a  translation  operator,  end  thus,  properties  a  to  d,  hold 

within  these  regions.   Property  e  holes  since  the  definition  of  V  arid 

* 

W  / 

n-1  gives 

f^^^'n  n  i"^n  n  1  i  1  >  for  i,j>l, 
*     ,  ,v        *  *  n-li,j-  n-li-1,  j-1        - 

I, J      ^,^j  1  i,J  i      n  i  1  i,j  ..   <^g  -r  u^  1  n^  for  i  or  j  =  1 

L 

Using  the.  induction  hypothesis  T*(V*  -!-  ^  W*  ,)<  EE  t.^  ,  (&fh^+h„)  =  fffh.+h„ 

n-1  —  Km   ij  Wn     A   B       A   B 

* 
since  T  is  a  Markov  operator.  This  result  does  not  depend  the  translation 

operator  characteristics  of  T  and  thus  holds  everywhere. 
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Crossing  from  one  region  of  the  optimal  policy  to  another  requires 

further  analysis.   In  crossing  the  boundary  at  which  the  optimal 

A  A 

X .  .  changes  from  X   to  0,  W   .  .  -  W        d.'oos  the  term 

^^  ^Vj  '  ^  ^n-1  i-M,j>  -  ^^'i  J  ■*■  ^  <-l  i,P    ^^°P^  ^^'   ^'   ^^^ 
expression  for  W  ...  At  this  point  the  term  '^ecoraes  negative  since 

the  optimal  X,.  is  ^ero,  bi:t  its  replac^-ment  by  zero  does  not  cause  an 

increase  since  previous  terms  of  this  kind  have  been  non-negative. 

All  of  the  other  terms  in  this  difference  are  positive  multiples  of 

terms  which  are  not  greater  than  their  counterparts  in  the  previous 

difference.  The  next  difference  looses  the  negative  of  this  term,  but 

since  the  term  itself  has  become  negative,  this  is  the  loss  of  a 

positive  contribution  to  the  difference.  When  increasing  the  first 

variable  causes  crossing  of  the  boundary  i,j*(t),  the  optimal 

X .  ^  changes  from  0  to  X  .  This  adds  the  term  X  [(V.^^,  +  3  W  ,  ,  ..,. 
ij  ij+1      n-1  i,j+l)  - 

v..  +  3W  T   ,]  to  the  first  difference.   Since  the  previous  test 
ij      n-i  ij 

B  B 

criterion  for  X..  is  negative  since  the  optimal  value  is  X.,  =  0, 

-  ^^t<-l  j.l  +  S  W^.i  ,.,_  ^j)  -  (V*^^^  +  9  W*.^  ^.,  j)]  may  be  added 
to  the  first  difference  producing  som.ething  larger.  .Rearranging  terms, 

one  has  X^[(V.  j^,  +  g  W^.,  ,^^j  -  ("Vl,  j+x  +  ^  Vl  i-1,  J+l>^ 

*   (1-^i-X^  [(V*_^  +  P  -/_^  y)  -  (V^_^_^  +  P  w;;_^  ..j_  p   plus  other 

terms  which  are  non-increaf^lng  either  by  the  induction  argument  or 

ir 

because  of  the  previous  discussion  of  crossing  the  i  (j),j  boundary. 
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In  place  of  these  two  special  terms,  the  previous  difference  one  has 

only  the  term  (i-ji)  (fV*     +  ^  W*  i  .  ,  .)  -  (V*  «  ,  +  3  W*  .  ^  _  \)  . 

1-1, J      n-1  1.-1,1      i-Z  j      n~l  i'2^y^ 

This  may  be  written  as   (l-p,-X  )  ^■  X  times  the  bracketed  terra. 

By  the  induction  (l-|jb-X)  tiuies  the  bracket  terra  ia  not  less  irhan  its 

counter  part  in  the  previous  expression.  Moreover,  also  by  induction, 

X^  ((V*  ,  .  +  3  W*  ,  ,  ,  J  -  (v''  .,,  +  3  W^  .,   ^  ,))  ie  not  less  than 
^  1-1, .1      n-1  1-1, j      i--ij      n-1  j--2,J' 

i  j+l      ik-i   i,j      .-),  j-<  ..      n-1  1-1,  j+1 
first  change  cannot  increase  the  first  difference.  Further  increasing 

the  first  variable  means  that  the  first  difference  will  have  both  the  terms 

-^^«<  i+1  +  ^  Vl  i.j+l>  -  ^^i,J  ^  ^  Vl  i.j^^hile  th.  previous  first 

difference  will  have  only  the  terra  X   ((V.  .^ -,  +  3  W  ,,:,.,)' 

i   j+1  n-1   i,j+l 

it 

-    (V       +   3  W         '.    ,)).     Eliminating   the  special   terra  from  the   previous 

■J  *  *j 

first  difference  can  only  decrease  its  value,  but  if  this  is  done,  the 

argument  just  completed  showsthat  what  is  left, is  still  not  smaller  than 

the  new  first  difference.   From  this  point  on,  the  positive  translation 

operator  agreement  is  again  valid. 

Having  proved  the  first  part  of  property  a  and  py  gymetry,  the 

first  part  of  property  b  now  consider  W   ,  ,  -  W   ,  ,  ,  as  a  function  of 

n  l,j    n  i-l,j 

j.  Again,  although  not  necessary,  it  is  possible  that  increasing  j  may 
cause  the  crossing  of  boi:h  boundaries,  Af^ain,  consider  each  one 

seperately.   In  croseing  the  boundary  from  the  region  in  which  the 

B      B 
Optimal  X  j  =  X  to  the  one  in  which  they  are  zero  the  first  terra  in 
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the  first  difference  to  be  dropped  is 

The  term  X^C'^^^,.,  +  8  W*_,  ,_,.,)  -  (V*^ .  +  g  W^  .  ^)]  is  left  by 
itself  and  cannot  ba  balanced  against  other  terms  using  positive  multiples 
of  non-decreasing  first  dif f^rerices  argumanc.  Since  thia  extra  terra  is 
positive,  just  ignore  it  and  what:  remains  Is  not  graater  than  the  true  first 
ference  and  is  not  less  than  the  previous  one.  Increasing  j  by  1,,  again:  iise  t-hf* 

device  of  writing  (l-p,-Ax^)  ((V*^  -r  p   w*  ^^^^)    -  (V*,^^  +  ^^'n<-l,^^^' 
The  (1-P.-X  )  balances  thfi  term  in  the  previoue  differenhs  corresponding 

B       ^ 
those  in  the  braclaet,  i.e. ,  the  same  term  with  j=j-l.  The  \  term 

equals  a  term  in  the  previous  first  differc.nce.   If  it  were  possible  to 

increase  j  further  without  dropping  the  term  X  ((V   .  -  4-  3  W  .  .  ,) 

•k  -fc 

-  (V^  .  4-  p  W   . .))  the  argument  could  be  in  difficulty  for,  although 

i  j      n  ij         ® 

other  terms  are  non-decraasing,  this  one  has  been  shown  to  be  non- 
increasing.   Fortunately,  this  term  must  drop  out  with  the  next  increase 
in  j.  Again,  everything  is  not  symetric  so  that  the  positive  multiplier 

argunjent  can  wortc  immediately.  Adding  ?*.  ((V,  -  .  +  B  W   ,  .  .) 

1-1, j      n  i-1, j' 

-  (V  »  .  T  +  0  vf   .  ,  .  ,)),  which  must  be  positive  since  the  optimal 

i-I  j-1      n  i-i,;j-l   '  '^ 

■a 

X,  ,  ,  .  is  0,  to  the  previous  difference  merely  increases  its  value. 
1-1,  J-1 

B      R 

This  allows  the  splitting  of  (1-ij,)  into  (l-p--X  )  -f  X  to  balance  terras. 

Thus,  the  new  first  difference  is  not  less  than  a  quantity  which  is  not 
less  than  the  previous  first  difference. 
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Next,  it  Is  necessary  to  explain  the  effect  on  W  .  ,  -  W  ^  .  ,  of 

n  1  j    n  1-1, j 

A     A 
crossing  into  the  region  in  which  the  optimal  \       =  X     because  of  an 

A  + 

increase   in  j.     The   first  effect   is   the  introduction  of   -X   ((V.    .  +   P  W      ,    ^    ,) 

i ,  j      n  - 1  i ,  j 

^  if 

-  (V     4-  p  W  , ,  .   )) .   In  the  first  difference  for  the  previous  J 
i  -  i ,  J      a***!  "  i- ,  J 

A       A  A* 

value,  split  the  1-p,  terra  into  (l-|j,-X  )  and  X  .  The  (l-p,-X^raultiplie8 

a  term  which  is  non-decreasing  in  the  successive  differences. 

X*((V*  j.i+  Sw;;,.  J.,)  -  (V*.i,j.i+  PW^^.j  j.j))  is  negative  since 

the  optimal  X^  ,  .  ^         j  .  ^  -i    ^   ^l     ..  j.j:^        t 

i-l>j  is  zero  and  its  loss  in  the  next  dLfference  only  serves 

to  increase  the  difference.   Increasing  j  by  1  means  that  the  two  differences 

will  both  have  (l-vL-X'S  ((V*.  +  P  W*_^  ^  ^)  -  (V*_^^  +  p  W*^^^  1-1,  j^^ 

terms  and  the  positive  multiplier  argument  works.  At  some  point,  j  increases, 

■  ( 

the  term  X  ((V.^,  ,  +  gw  ,  .^  ,  .)  -  (V.,  +  PW  .  .  J)  appears  and  it 

i-l-l,j      n-1  5+1, J'     ij      n-1  i,j'' 

causes  an  increase.  Further  increases  in  j  are  permissible  under  the 

positive  translation  operator  argument.  Thus,  property  a  is  justified 

completely,  and  by  symmetry,  so  is  property  b. 

Next,  properties  c  and  d  need  to  be  verified.  As  i  increases  and 

(W   .  .  -  W   .  ^  .)  -  (W   .  •,  ,  1-  W   .  o  4  t)  begins  to  cross  the 
n  i,j    n  1-1, j      n  t-l,j-l   n  i-2,1-i    ^ 

A  A 

boundary  at  which  the  optimal  X.^  changes  from  X  to  0,  the  first  change 

is  the  loss  of  a  tern  of  the  four  ^   ((V..,  ^  +  W  ,...,,)-  (V.  .  +  W  -  .  .))  . 

i+l,j  n-ii+i,j  i-,j  n-1   i,J 

A     *  * 

V?hen   this   happens,    the   corresponding   term  -X  ((V .    ,    ^   +    pW     ,    ^.    ,)    - 

^        ^1  j-1      n-1  ij-1 

*  * 

(V,  ^  ,  .  4-  3  W  1  -  -J  4  i))n^*'y  or   may  not  disappear,  but  since  it  is 

negative,  it  cannot  make  the  difference  of  differences  positive.   In 

what  remains,  there  are  only  positive  multiples  of  negative  differences 

of  first  differences.  The  second  term' must  disappear  with  an  increase 

A  A 

in  i,  since  when  the  optimal  X. .  '^  0  so  must  X, .  -  =0.  The  positive 

ij  ij-1 


< 
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multiple  agreerrieiit   requires   corabintng   terms   so  that   one  has 

^   ^  -J  A 

"  '*^^={  9  ■!  1  "^  ^  '"^  1  '  •?  •»'>^-  with  1-u-X  >  0  since  it  is,  of  course, 

a  probability.  The  next  term  which  is  dropped  is  -X   ((V.  ,  -  p  W  n  ^  .) 

1 9  J      n-i  1 , J 

-  (V.  ^  ^  -1-  3  W  •,  .  -f   ))  because  it  bec':^rae,<5  positive.  This  change 
serves  to  reduce  the  difference  of  differences  below  the  already  negative 
result  that  the  positive  multiplier  argument  gives. 

Continuing  the  discussion  of  this  difference  of  differences,  it 

is  necessary  to  examine  what  happens  as  the  boundary  at  which  the  optimal 

B  B 

X^ ,  changes  from  X  to  0.  The  first  effect  on  (W   ..  -  W   .  ,  ,) 
ij  n  ij     n  i-l,j' ^ 

'K   i-lj.l  -  ^n  i-2,J-l>  ''    ^^^^  ^^'^  ^^^^  -^^  ^<^!-l,j+l  ^  ^  Cl  i.l,j^l> 

-  V*  .  4  +  ^  W   ^  .))  is  eliminated.  This  gives  no  difficulty  since 

1- i , J      ni- i , J 

this  happens  only  when  the  term  is  positive.  Dropping  a  positive  term 
merely  makes  the  result  more  negative  than  the  already  negative  result 

of  the  positive  multiplier  argument.  The  next  loss  is  not  unique  and 

B   ^         i( 
might  happen  simultaneously.   First,  the  term  considers  X  ((V.  -+PW  ,.^.) 

i~z,j     n-l  i--i,j 

•k  * 

-  (V.  ^  .  ,+  3  W  1  .  ^  .  ^)).  Either  this  drops  out  either  simultaneously 

with  the  first  loss  or  on  the  next  increase  of  j  because  the  form  of  the 

boundary  i  (j).   If  this  drops  out  with  the  first  loss,  then  the  only 
B  ,     .,,     ,  ,    .  ,    ,Bt 


remaining  X'  terms  can  be  collected  bv  giving  X  [({'V.  ^.  +  3  W  ,  ,  .,-) 

•  Xj  j+i      n-l  i,jTi 

-  (V*    ,  +   3  if   ,  .  ,))  -  {(V\'  ,  .4-  3  w''  ,  .  _  J  -  (Vf  ,  ,  -  +  3  W*  ,  ,  ,  ,  ,..-1 
i,j      a~i  i,y       i-i,J     n-l  1-1, j      i-l,j-l      n-l  i-l,j-l))J. 


This  is  negative  as  a  result  of  property  d  of  the  induction  hypothesis. 

The  term  }-^'((y*    ..  ,  4  ^  W*  ,  ^  ...  -  (V*  ..  +  ^  W*  ,  .  J)  must  drop  out 

:i,j-»-i      n-..  i,j+i;     i,j      n-l  i,j 

with  the  next  increase  of  j.   If  it  drops  out  simultaneously  with  the 
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loss  of  the  first  term,  one  can  have  left  -  \  [((V^  ,  .  +  P  W  ,  ^  ,  J 

i-l.j      n-1  i-l.J 

^^i-l,j-l     ^n-i  i-l,j-r^   ^^  i-2,j   ^  ^n-1  i-aj'' 

-  (^\    „  .  .  +  3  W  ))].  Th-  term  in  the  bracket  is  positive 

since.it  is  the  difference  of  differences  which  are  now  decreasing.   Thus, 

multipled  by  ->.  makes  the  pioduct  non-positive.   The  remaining  terms 

are  negative  since  they  are  positive  multiplp.s  of  non-positive  differences 

of  differences  by  hypothesis  c.   If  the  first  and  the  next  two  terms  drop 

out  simultaneously,  the  last  remaining  X  term  is  -X  [(V,  ,  .  +  3  W  ,  .  ,  .) 

i-l,J      n-1  i-l,j 

-  (V,  ,  .  1  +  3w   -,.-,,  i^'»  but  this  is  negative  since  the  term  in 

i-l,j-i      n-l  i-i,j-i 

the  bracket  is  positive.   This  term  also  drops  out  at  the  next  increase 

B 
in  j  and  none  of  the  X  terms  are  left.   This  completes  the  proof  of 

property  c  and  the  symmetric  counterpart  property  d. 

This  completes  the  induction.  In  the  analysis  the  only 

terms  involving  the  arrival  rate  which  was  known  to  be  changing  at  each 

boundary  were  discussed.   It  is  possible  that  in  some  cases  the  two 

boundaries  are  crossed  simultaneously.   This  presents  no  problem.   The 

A         B 
only  possible  problem  comes  when  l-p,-X  and  1-|.;."X  are  used.  If  necessary, 

A  B 

however,  k. -X  >  0  dn6   V-.,^    -    X  >  0  cnn  be.  useu  with  k  -fk  «=l-u,  since 

A   R 

1  -  ^.-  •  -  -  '  IS  .^.  t>r  oba  t^  I  i  t  v 


DynarTtics 

Dynamic  properties  of  t^iis  analysis  seem  somewhat  more  difficult 
to  establish  than  the  results  on  the  form  of  the  optimal  n  period  policy 

One  result^ which  is  relatively  easy,  is  that  there  exist  bounds  i* 
and  j*  such  tbst  for  all  n  the  optimal  arrival  rates  are  X^ .  =  0  for 

i  >  i*  and  X^  =  0  for  1  >  1* .   The  argument  for  this  is  that  the  n 

ij  -  ■ 

period  profits  can  be  written  as'  a  sum  of  profits  in  each  of  the  n 
periods.   For  large  i  the  first  period  profit  will  h^  less  in  state 


i 
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i  -f  1  than  It  is  in   scate  i.   For  at  least  the  next  i  periods,  all 
sample  paths  starting  from  i  4-  1  will  have  at  least  as  large  an  inventory 
of  A  parts  se  will  those  from  snate  5.,  and  thus,  the  profits  starting 
from  i  +  1  will  not  excead   those  starting  from  i  for  this  interval. 
Equality  can  occur  i.t  at  some  txvnt  the   hiRtory  generated  starting  from 

i  +  1  is  in  n   statf  k,m  in  which  ^^'^  o>timal  V    is  0,  but  the  ootimal 

A        A 
X.  .    is  X  .   In  this  care,  some  of  the  histories  Titarting  from  i 

will  have  the  same  level  of  A  inventory  ac  the  corresponding  ones 

starting  from  i  +  1.  From  this  time  on,  there  will  be  a 

one-to-one  match  of  the  remaining  possible  history  of  the  process. 

On  the  other  hand,  there  will  be  histories  in  which  the  number  of  A  items 

on  hand  is  always  1  more  starting  from  i  +  1  than  is  the  cast  starting 

from  i  until  the  level  0  is  reached.  At  this  time,  the  histories 

starting  from  i  +  1  have  a  profit  which  is  p^  -  h  higher  than  those 

starting  from  i.  For  the  remiaining  time  until  the  horizon,  the  possible 

histories  may  ssptirate,  buc  the  difference  in  profits  is  bounded  by 

s>4P  !w  .,  X  -  W  o  <  1  I  .~  ^"^a'^^w  ^y  induct 5  on.   Thus,  the  difference 
in  the  worth  of  starting  in  i  +  1  and  starting  in  i  is  not  greater  than 
-h,  -h   fj  [g'i'h.+h  1  which  can  be  made  negative  for  i  sufficiently  large. 
Clearly,  when  \'i    ,  ^^     .    -  Vf  .    .  <  0,  the  optimal  X  ,,  ,  ^  =  0.  This  also 
easily  establishes  that  if  n  <  i,  X* .  =  0.   In  n  periods,  no  more  than 
n  type  A  items  can  be.  used,  and  thus  profits  can  only  be  decreased  by 
adding  a  type  A  item  at  any  time  during  the  n  periods. 
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The  bounds  i*>  j*  just  discussed  can  be  used  to  say  that  there  is  a 
finite  state  finite  action  problem  for  which  the  recursive  optimal  dis- 
counted expected  profit  functions  W^^j.  ^   ^^^-   equal  to  the  infinite  state 

^ni  1  ^^^^  (i»j)  j5  (i*>j*)-   This  follows  iinmediately  from  the  transition 

A  8 

operator,  which  guarantees  that  if  }^ ^   =0  for  i  >  i*,  and  X  =  0  for  j  <  j* 

and  all  n,  then  Wni,j  depends  oni.y  on  W^^i^j  for  i^  <  i*  and  j  <  j*  and 

e 

m  <  n.   This  dependence  is  the  same  for  W  and  W„. 
•-  n      n 

As  far  as  the  behavior  of  W  as  n  approaches  infinity  is  concerned, 
the  fact  that  the  system  evolves  according  to  a  Markov  chain  having  positive 
probability  of  only  finite  changes  instate,  and  the  fact  profits  are  dis- 
counted by  3  <  1,  combine  to  guarantee  convergence  for  all  finite  states. 
A  standard  contraction  argument  easily  establishes 

'  nfl  i^    ni^j  '  —   '  nl,j    n-1  i,j' 

This  unfortunately  does  not  provide  a  characterization  of  the  sequence 

of  optimal  policies.  There  is  one  obvious  result  which  can  be  supported. 

A  A 

If  X.  ,  =  0  and  X.  -  ,  .  =  0  in  the  optiiTtai  policy  for  period  n,  then 
ij  1-1, J-1  r  J  r 

X.   =  0  for  period  n  +  1.   In  this  case,  the  test  criterion  is  strictly 
negative. 

A      n^l  i+i,j    nfl  i,j     A     ^   ^^A    ni+l,j    ni,j 


^   A  n  i,j-l    n  1-1,  j-1   —   A 


A  A 

This  implies  that  the  region  in  vrhich  X.\  should  bo  X  cannot  expand  by 

more  than  1  in  each  period.   If  i  is  sufficiently  large  that  the  optimal 

X,  .  =  0  for  all  periods  up  to  n  and  states  which  can  be  reached  from  i. 
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n4*2 
then  this  criterion  behaves  like  -h  (1-3   )/(l-3)  which  is  monotonia 

decreasing  in  n. 

What  is  both  more  interesting  and  more  difficult  is  the  behavior 

A  A 

in  the  regions  in  which  X.   should  be  \  .  So  far  no  counter  example  has 

^  J 
been  found  for  the  hypothesis  that  the  optimal  X   are  non-decreasing 

functions  of  n.  A  proof  that  this  must  be  so  involves  showing  that 

A      nfl  i+l,j      n+1  i,j'  —     A      n  i+l,j      ni,j 
whenever  the  right-hand-side  is  positive.  This  is  equivalent  to 

nfl  1+1, j    nfl  i,j  —   n  i+l,j    ni,j 
when  the  right-hand-side  is  >  h. .  All  that  is  easy  to  argue  is  that 

(-h.  +  W   .^T  4  -  W  .  .)  >  0— ^(W_^,  .^,  .  -  W_^,  .  ,)  >  0 
^  A    n  i+I,j    ni,j  —    ^^   n+1  i+l,j    n+-l  i,j  — 

This  is  an  immediate  consequence  of  the  recursive  definition  of  W  - 
which  permits  this  difference  to  be  written  as  a  sum  of  positive  terms. 
To  say  more  than. this  requires  a  stronger  inequality  than 

ni,j-l    n  i-1,  j-l-^  -   n  i-i-l,j    ni,j^ 

v;hich  was  shown  earlier.  A  stronger  version  does  not  hold  everywhere. 

A 
When  the  optimal  X.  ^    -  =  0,  there  can  be  equality  especially  for  small  n, 
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Small  Intervals 

The  natural  interest  in  differential  equations  for  queueing 
systems  resulting  from  the  M/M/1  analysis  raises  the  question  of  what 
happens  if  the  interval  size  is  reduced.  To  examine  this,  one  needs  to 
reformulate  the  problem  somewhat.  Following  the  usual  approach,  the 

probabilities  of  the  three  possible  state  change  events  are  re- 

A       B 
defined  as  X  A  t,  X  At,  and  p,  A  t.  The  holding  cost  can  be  re- 
defined as  h.  and  h  per  unit  time  respectively.  The  cost  of  holding 
one  type  A  item  for  A  t  is  now  h  A  t.  Finally,  the  discount  factor  has 

A 

to  be  modified  to  be  1  -  ct  A  t.  The  criterion  for  choosing  the  n  +  1st 
X   in  these  terms  is 

-h^it+  (1-  aAt)  (W^^^j  J  -W„  ._^) 

Since  this  is  an  affine  function  of  At,  it  changes  sign  at  most  once  as 

At  goes  to  zero.  This  guarantees  that  shrinking  At  will  not  produce 

oscilations  in  the  transition  probabilities  which  would  make  the 

limiting  differential  equation  meaningless.  At  the  moment,  this  differential 

equation  is  not  of  computational  interest.   Perhaps  the  most  interesting 

aspect  of  the  formulation  leading  to  the  differential  equation  occurs 

in  the  limit  as  the  time  paraneter  gets  large.  To  the  accuracy  of  At 

'V<"Hj»^ij>ij  "  VijAt  {-h^i  -  hgj  +  |jg  e^j}  At. 
and  the  limiting  W  for  any  At  satisfies 


0  -  max     (V,,  At  (TCXf.,  X?.)  -  I)  At  W) 
^A    >B     •*-J  ^J     -^ 


XA    X&. 
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As  usual,  in  queueing  processes,  the  Lt   can  be  factored  outj  both  the 
limiting  W  and  the  opuiraal  policy  do  not  depend  on  the  size  of  At. 
This  also  is  the  limit  cc  which  tne  solution  to  the  differential  equation 
must  converge.   In  M/M/1,  an  inter pre tat icn  of  the  unimportance  of  At 
is  that:  the  limiting  probabllilties  depend  only  on  ratios 
of  transition  prooabilities .  not  their  absolute  values.  This  is,  of 
course,  also  true  here  for  the  probabilities.   It  is  not  true  that  an 
optimal  stationary  policy  depends  only  on  ratios  of  the  cost  parameters. 
The  scaling  poesibility  here  is  among  the  holding  costs,  h.  and  h   and 
the  expected  revenue  per  unit  time  given  that  the  operation  is  working 
|jg.  Thus,  the  absolute  value  of  jj.  does  become  important.  To  eliminate 
this  one  must  factor  from  the  function  to  be  optimized  iiAt  in  which  case 
h. /p,  and  h  /p.  the  expected  holding  cost  during  an  assembly  will  be  the 
holding  cost  parameters. 

Dependence  on  jj. 

In  many  situations,  not  only  Is  it  possible  to  control  the  inputs, 
but  also  it  is  necessary  to   choose  the  itian  and/or  equipment  to  perform 
the  assembly.   In  this  model,  this  is  f^:quivalent  to  choosing  the  com- 
pletion probability  li.  This  choice  is  made  once  when  the  system 
begins  operation.   In  general,  this  decision  depends  on  both  the  sequence 
of  arrival  probabilities  and  the  initial  conditions.  The  combined 
decision  problem  has  the  form 


^"^   {(   Po,  max  W  (D))  -  G(^)) 
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In  this  expression  D  is  an  infinite  sequence  of  decision  functions  D-j. 

A       B 
each  one  of  which  specifies  X,  .   and  \.  .   for  each  state  i j .  The  function 

C(p,)  is  a  cost  for  choosing  \i,   as  th-' assembly  probability.  The  first 

bracket  represents  the  scalar  product  and  is  the  expectation  with  respect 

to  the  initial  conditions  of  the  discounted  expected  return  over  an 

infinite  horizon  using  policy  D  for  a  fixed  value  jj,  .   In  many  situations 

the  set  of  values  9  from  which  |j,  mist  be  chosen  is  a  finite  set  and 

a  maximum  will  exist.   If  the  number  of  values  in  9  is  small,  it  will 

be  feasible  to  solve  the  problem  by  enumeration  of  the  |j,  values.   If 

C(iJi)  is  quite  irregular,  such  a  procedure  may  be  the  only  possibility. 

From  a  modeling  point  of  view,  even  if  9  is  interval  [0,|A  ],  it  is 

probably  permissible  to  approximate  6  by  a  finite  set  containing  only 

a  few  values  since  the  entire  model  is  an  approximation  anyhow.  From 

any  point  of  view  other  than  immediate  implementation,  this  resort  to 

crude  numerical  methods  has  little,  if  any,  appeal.  The  problem  is  to 

develop  properties  of  max  W  (D)  as  a  function  of  u,  and  assumptions  on 

D   M* 

the  form  of  C(m>)  which  will  insure  that  a  more  efficient  sequential 
search  procedure  than  enumeration  can  be  used  to  solve  the  problem. 
The  situation  of  being  able  co  partially  characterize  the  optimal  policy 
of  a  sequential  decision  process  2nd  possesion  of  relatively  efficient 
computational  means  for  finding  it  is  typical  of  the  analysis  of  many 
^ueueing  systems.  As  in  this  example,  this  should  not  be  the  end  of 

the  analysis,  for  real  problems  have  both  design  (selection  of  p.) 

A       B 

and  control  (selection  of  >,.  .  and  X^  ,  for  each  decision  time)  aspects. 
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For  applications,  it  will  generally  be  much  more  difficult  and  expensive 
to  rectify  errors  in  design  than  to  improve  control. 

An  important  question  to  answer  is  whether  an  optimal  p,  exists. 
Unfortunately,  the  dependence  of  the  functions  W  ,  .  on  pi  is  not  as  easy 
to  describe  as  one  might  hope.  Consider  the  derivatives  d  W  ,  ,/dii.  First 


dW, .  .   dV,  , 


«  —w 


d|ji 


d  ^ 


=  (g-^A-Hig)  Hj 


^\03        ^V,j 


=  0 


dM.^     d^' 


These  one  period  functions  are  extremely  well  behaved,  but  examination 
of  dW^ .  ./dp.  gives  a  more  complicated  picture.   For  the  second  period 
only  X  .    for  j  >  0  and  can  be  positive.  The  function  is 


/C-\i-hgj+M.(g4-h^+hg)]  (l-ffi)  +P.3  (\+\) 


i>l,  j>l 


[-h   -h  j+iA(&fh  +h  )]      (l+P(l-p,))H-   M,3   (-h_(j-l))        i=l   j>l 


A     B 


B 


"2i,j  <      ^-^A^"V^^^'W^    (l+3(l-ti))  +P.3   (-h^(i-l))  j=l    i>l 


-V!o"^^   (1+(1-X^^)3)  +3X^^   ('\i'\+ \^(^h^+h^))      i  >  0  j=0 


iO   ^     A       B 


i,j   «,0,0 
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The  first  derivatives  are 


dW 


ILl 


d|j, 


(&<-h^+hgO  (1+6)  +  PCV^ig) 


(&4-h^+hg)  (l+3-2p.^)  4-  (h^+b^) 


I   -  ^A 


I  P  Xoi  ^^^^^; 


Oj 


B 


^10  <^V^> 


v° 


i>l  J>1 

i«l  j>l 

i>l  j°l 

i=0  j>0 
i>0  j=0 
i=0  j»=0 


A  B 

At  the  points  at  which  X..  and  X.^  change  values  the  derivatives  are 

OJ      lO 

undefined.  The  test  criterion  for  X^ .  is 


-V^^^i  ij  -  ^ioj>  =  -V^(<^^^V^B>  -V 


which  is  negative  until 


g 


A 
Thus,  for  small  ^i,,  X„   should  be  zero, while  for  large  values  it  becomes 

A 
X  .  This  makes  W^^  ,  piecewise  linear  and  convex.  W^ .  ,.  is  similar. 

W^,  .  and  W^  ^  are  concave  increasing;  while  the  remaining  functions 

W«,  .  for  i>l  j>l  are  increasing  and  linear  in  p,. 


-24- 

For  the  general  case,  one  has,  by  induction,  that  the  functions 

W 
nfl  i,j  are  continuous  in  \m   since,  for  any  continuous  function, 

A 
G(|jl),  X  (inax(0,G(iJ,)))  is  continuous.   For  the  intervals  in  which  the 

derivatives  are  defined 

'^,^1  1.1    '^^4l-^!l>  ,  ^  f^«(^-«   ^n  1-1.  1-1 
dp,  p,  dp, 


dp, 


A  B 

ij    n  i+1 ,  T     ij    n  i,j-H   + 

dp,  dp, 


n  i-l,j-l  ni,j 


A  A 

As  in  the  case  of  W,^,^  .  when  X,  .  changes  from  0  to  X  ,  the  derivative 

20,  J  ij  '^ 

of  W   . ,    .    .   experiences   a  positive  jump.      Since  W    ..-    ^    -  W      ,    ,   must  be 
n+-l   i,j        '^  j^v^;:i.uuj.,,c  J      r  ni+l,j  n  i,j 

n 

increasing.  The  same  holds  for  X   .  When  the  derivatives  exist,  they, 
must  be  non-negative.  Under  the  induction  hypothesis, 

<1W  .,  ,  .  >  dV(X^   X^.)  +  3|ie(ij)   (W„  .  T  ,  T  -  W  ..) 
"^1  ^»-1        ^J   ^J  n  i-l,j-l    nij 

dp  dp, 

Since  W  .  .  -  W    ..   .  has  already  been  shown  not  greater  than  gf h  +h  , 
'^^jj"''-'"^>J'"^  AB 

the  right-hand-side  must  be  non-negative.  These  derivatives  inherit 
discontinuities  from  those  of  W  .  All  discontinuities  give  positive  jumps, 
if  for  any  n  X   changes  only  once  from  0  to  X  and  similarly  X   changes 
at  most  once. 


-25- 

Inductive  analysis  can  show  that  the  test  criterion  is  non- 
decreasing  or  equivalently  it  is  continuous  and  d(W  ,,  ...  ^,.      -  dW  . ,  ,    ..  ,  , 

^  n^-l  i+l,j/d|j,     trfl  i,j)/dp, 

is  non-negative  when  it  is  defined.   Continuity  follows  the  continuity  of 
W   .  For  one  period,  the  derivative  is  zero  except  for  i=0  j>0  when 
it  is  positive.   The  gener-'^l  case  is 

dp,  dp,  dy, 


dW  -  dW 

+  a  M.e(i+l,j)   °^ni  .1-1   -  PM.e(i,J)    n  i-lJ-1 

d|j,  d|ji 


^^^^-^i+l,j"^i+l,j"^'^^"^^'^^^  ^^  ^-^^'^ 
^'-'    ^'J  dp. 


+  3  XA      ^^n  i-f2,1   -  3X^  .   ^^n  i-fUJ 
^-"^'^  — dT^      ^'J  — dT^ 


B      dW  Q  ^B    dW 


dy,  dp, 


3s(i+l,j)  W      +  3e(l+l,j)  W      +  0e(i,j)  W 

nii-ijj  ni,j-i  ni,J 

.pe(i,j)  W 

ni-i,3~l 
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The  terras  involving  derivatives  of  W  make  no  positive  contribution  due 
to  the  induction  hypothesis  when  e(i,i)=l  and  the  decisions  are  the  same 
in  i+l,j  and  ij.   Similarly,  if  e(i+i , j)  and  G(i,j)  are  positive,  the 
last  four  terras  must  have  a   non-negative  sum  because  of  the  properties 
of  W  previously  shown.  There  are,  of  course,  other  special  cases  to 

■D  R        Ti 

be  considered.  First,  it  may  happen  that  X      =  \     and  X,  .  =  0, 

i**'i>J  i-jj 

but  this  does  not  lead  to  a  negative  contribution.  When  X,  ,  .  changes 

i+  i ,  J 

D  R  ■n 

from  0  to  X  ,  there  is  a  positive  jump,  and  when  X   becomes  X  ,  a  smaller  negative 

^  J 

A  A  A 

jump.     When  X. ,   becomes   X     and   X. . ,     .   =  0,    then   there   is  a  negative  jump 

of  -X   (dW   ,,,    wj      -   dW    .    wj   )j    t>ut   this    is  nullified  by   the   positive 
ni+l,j/d|j,  ni,j/d^i   ' 

term  (l-X^^^    ,,   -  ^c(i+l,j))   <«'„,^i,3/d^  -   d-X^.j   "  V'^d.J))   ^ni,i/6^- 

A  A 

When  X.  .  .  becomes  X  ,  there  is  a  positive  jump.  The  possibility  of 

«(i»j)  *  0>  e(i+l>j)  =  1  only  contributes  a  positive  addition  tb  the 
derivative  terras,  for  -^wiW  .  . /dy  is  replaced  by  the  larger  value  zero. 

In  the  last  four  terms,  this  possibility  produces  -W  . ,  , .  +  W  .  .  ,,  but 

■^  ni+lj    ni,j-l' 

by  the  previous  induction  this  must  be  not  less  than  -g-h. -h  ,  which 

A      B  A    B 

is  the  negative  of  dV(X.,,  "^-j.!  x^^^\^   "  ^V(X   ,X,  .)/d|j,  in  this  situation. 

Thus,  inductively  dW  .,  .,,  ./du  -  dW  , ,  ^  ./da  is  non-negative  when  it 

n+-l  i+l,j   '^     n+-l  i,j   "^         ° 

is  defined.  This  means  that  there  is  at  most  one  value  of  [i   at  which 

A  A  A. 

X.  .  will  change  from  0  to  X  and  it  will  remain  X  for  all  higher  values 

■a 

of  \i..     A  symmetric  argument  applies  to  X   . 

Optimal  ]i, 

The  result  just  obtained  guarantees  that  any  stage  the  optimal 

A       B 

X. .  and  X^  .  are  well  behaved.  The  monotonicity  of  W  ^  ^  in  u,  is  not 
ij      ij  ^     ni,j    ^ 

a  strong  enough  property  to  guarantee  that  there  will  be  a  unique 
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optitnum  |ji  for  the  design  problem  posed  in  the  previous  section.  Much 
more  is  necessary  if  one  wishes  (P-^,W   )  -  C  (p»)  to  be  unimodel.  The 
obvious  disirable  property  of  concavity  does  not  hold  for  W«.  ..  The 
•difficulties  of  the  analysis  of  this  system  are  typical  of  problems 
of  design  and  control  of  Markov  systems,  especially  when  the  natural 
state  space  is  two  or  highei  in  dimension. 

Even  if  it  were  possible  to  show  that  under  reasonable  conditions 
there  is  an  optimal  |j,,  there  still  retoains  the  problem  of  finding  this 
value.  An  iterative  procedure  which  approximately  solves  the  dynamic 
programming  problem  for  the  optimal  control  for  each  value  of  jj,  and  searches 
among  these  solutions  for  an  optimum  has  little  to  offer  other  than 

its  feasibility.  What  is  needed  is  an  iterative  procedure  which  will 

A       B 
pick  a  sequence  |i(k) ,  \. .(k),  X. .(k)  j  k=l.,.  which  will  converge  to 

an  optimum  if  one  exists,  without  the  necessity  of  ^,(k)  being  constant 

for  large  intervals  of  k  values. 

In  the  study  of  these  systems,  the  author  has  engaged  in  soma 

rather  extensive  numerical  vrork.  Unfortunately,  the  results  of  this 

work  are  not  in  a  form  that  they  can  be  presented  as  yet.   Perhaps  the 

most  striking  result  so  far  is  the  very  small  number  of  states  which 

are  ergodic  in  these  systema .   In  most  cases  so  far,  optimal  queue  sizes 

have  been  under  10,  and,  moreover,  rrjany  fewer  than  the  corresponding 

maximum  of  121  states  have  been  ergodic.  Although  it  is  easy  to 

introduce  further  complexities,  which  will  cause  any  numerical  analysis 

♦:o  tax  the  power  of  a  computer,  it  is  striking  how  much  of  the  imagined 
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dlfficultles  disappear  in  calculation.  The  results  presented  here 
really  constitute  an  "academic"  exercise,  for  they  all  had  strong 
support  from  calculations  before  the  inductions  had  been  completed. 
Only  an  "academic"  could  afford  to  ask  are  these  properties  always 
true  before  considering  what  happens  when  the  structure  of  the  problem 
is  changed. 
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